A Private web server must not respond to requests from public search engines.

Overview

Finding ID	Version	Rule ID	IA Controls	Severity
V-2260	WG310 IIS6	SV-28797r2_rule	ECLP-1	Medium

Description
Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web site content. In turn, these search engines make the content they obtain and catalog available to any public web user. Such information in the public domain defeats the purpose of a Limited or Certificate-based web server, provides information to those not authorized access to the web site, and could provide clues of the site’s architecture to malicious parties.

Description

Search engines are constantly at work on the Internet. Search engines are augmented by agents, often referred to as spiders or bots, which endeavor to capture and catalog web site content. In turn, these search engines make the content they obtain and catalog available to any public web user. Such information in the public domain defeats the purpose of a Limited or Certificate-based web server, provides information to those not authorized access to the web site, and could provide clues of the site’s architecture to malicious parties.

STIG	Date
IIS6 Site	2011-10-03

Details

Check Text ( C-30022r2_chk )
1. Open the IIS Manager > click on the web site being reviewed. 2. In the right hand pane look for a file named robots.txt. 3. Open the robots.txt file. 4. Ensure the following entry exists in the robots.txt file: User-agent: * Disallow: / If the robots.txt file does not exist or the entry above is not contained in the robots.txt file, this is a finding. NOTE: If other restrictions are in place to limit search engine access to the web site, and it meets the requirement, this would not be considered a finding.

Check Text ( C-30022r2_chk )

1. Open the IIS Manager > click on the web site being reviewed.
2. In the right hand pane look for a file named robots.txt.
3. Open the robots.txt file.
4. Ensure the following entry exists in the robots.txt file:

User-agent: *
Disallow: /

If the robots.txt file does not exist or the entry above is not contained in the robots.txt file, this is a finding.

NOTE: If other restrictions are in place to limit search engine access to the web site, and it meets the requirement, this would not be considered a finding.

Fix Text (F-32685r1_fix)
Establish a means to restrict search engines on the private web site.